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SECTION  I 

introduction 

Determination  of  student  performance  level  and,  subsequently,  decisions 
to  either  continue  or  stop  training  have  posed  a  perplexing  problem  for 
instructors  and  training  managers.  This  problem  is  especially  troublesome 
for  instructors  and  training  managers  providing  pilot  training.  In-flight 
training  for  pilots  requires  considerable  resource  expenditures  involving 
both  highly  skilled  human  resources  as  well  as  sophisticated  equipment. 
Training  Is  generally  accomplished  by  a  one-on-one  instructor-student  rela¬ 
tionship.  Thus,  training  continued  beyond  established  training  objectives 
is  costly.  However,  termination  of  training  before  the  student  pilot  achieves 
the  skills  required  of  him  in  the  precise,  ttHatlon  environment  Is  eleo  highly 
undesirable. 

Rankin  and  McDaniel  (1980)  proposed  a  Computer  Aided  Training  Evaluation 
and  Scheduling  (CATES)  system  for  achieving  Improvements  In  the  precision  of 
proficiency  judgments  and  In  determining  student  proficiency  during  In-flight 
training.  This  method  provides  a  computer  managed,  prescriptive  training 
program  based  on  Individual  student  performance.  The  CATES  system  uses  a 
proficiency  grading  system  developed  by  Browning,  Ryan,  Scott,  and  Snode 
(1977).  These  grades  are  then  evaluated  as  they  are  awarded  using  a  sequen¬ 
tial  sampling  technique  as  a  means  for  making  statistical  decisions  with  a 
minimum  sample  Introduced  by  Wald  (1947).  According  to  Rankin  and  McDaniel 
(1980),  the  conceptual  CATES  decision  model  augurs  well  with  the  present 
system  of  Instructor  judgments.  Nhet  remains  Is  to  assess  the  efficacy  of 
the  CATES  decision  model  using  actual  data  and  to  determine  from  this  assess¬ 
ment  If  the  CATES  system  offers  seme  practical  advantage. 

PURPOSE 

The  objectives  of  this  study  are  twofold.  The  first  objective  is  to 
compare  the  efficacy  of  the  CATES  system  with  the  present  system  of  "human 
judgments"  for  performance  assessment  In  flight  training  with  regard  to: 

'f 

m  efficiency  in  reaching  decisions 

•  quality  of  decisions. 

Increased  efficiency  In  reaching  training  decisions;  e.g.,  reduced  Informa¬ 
tion  requirements  to  determine  when  to  stop  training,  could  result  In  signif¬ 
icant  reductions  in  training  costs.  Increased  quality  of  training  decisions 
would  produce  a  more  effective  utilization  of  training  resources  and  reduce 
the  risk  of  Incorrect  decisions;  e.g.,  the  decision  Is  made  to  stop  training 
when  additional  training  Is  needed.  The  second  objective  Is  to  demonstrate 
that  the  CATES  system  can  be  used  with  some  advantage  In  an  actual  flight 
training  program. 

ORGANIZATION  OF  THE  REPORT 

In  addition  to  this  Introduction,  four  sections  and  six  appendices  are 
presented.  Section  II  presents  the  development  of  the  statistical  decision 
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model  used  by  the  CATES  system  and  results  of  other  evaluations  using  mathe¬ 
matical  models  versus  human  models  in  decision  making.  Section  III  presents 
the  method  used  for  comparing  the  CATES  system  decision  model  with  the  present 
system  of  decision  making  and  the  operational  definitions  used  In  this  evalua¬ 
tion.  Section  IV  presents  the  results  and  comparisons  of  efficient  use  of 
information  In  reaching  decisions  and  the  quality  of  the  decisions  as  evidenced 
by  performance  on  a  final  flight  evaluation.  Section  V  presents  a  discussion 
of  the  results  and  formulates  conclusions  based  on  the  findings  with  recommenda¬ 
tions  for  further  applications  of  the  CATES  system. 

Appendix  A  contains  a  description  of  the  Wald  Binomial  Probability  Ratio 
Test.  Appendix  B  is  a  listing  of  the  tasks  and  respective  task  parameters 
that  were  used  in  this  evaluation.  Appendix  C  contains  the  tasks  used  to 
evaluate  decision  efficiency  as  a  function  of  difficulty.  Appendix  D  contains 
a  sample  grade  card  used  for  data  recording.  Appendix  E  contains  a  copy  of 
the  Naval  Air  Training  and  Operating  Procedures  Standardization  Program  (NATOPS) 
Evaluation  Worksheet.  Appendix  F  contains  the  mathematical  equation  used 
for  estimating  trials  to  a  training  decision  for  the  CATES  decision  model. 
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Other  approaches  for  determining  level  of  proficiency  were  investigated. 
One  such  approach  was  to  arbitrarily  asses*  proficiency  as  being  reached 
after  the  student  had  demonstrated  performance  to  standards  on  two,  three, 
or  four  successive  trials.  Such  an  approach  was  used  in  the  Initial  Entry 
Rotary  Wing  Flight  Training  Program  by  the  Army  (USAAVNC  Evaluation  Team, 
1979).  The  logic  of  such  an  approach  was  appealing;  however,  arbitrary  selec¬ 
tion  of  the  number  of  proficient  trials  needed  to  demonstrate  proficiency  do 
not  account  for  variability  in  student  performance,  task  difficulty,  and 
variability  in  instructor  ratings  (Rankin  and  McDaniel,  1980).  Also,  both 
the  approach  used  by  Browning,  et  al.  (1977)  and  the  USAAVNC  Evaluation  Team 
(1979)  required  training  protocols  that  include  initial  and  final  levels  of 
proficiency  to  make  accurate  performance  determinations.  Neither  approach 
could  accommodate  situations  where  only  a  small  number  of  training  trials 
are  given  or  where  there  are  wide  differences  in  learning  rates  of  students. 
Further,  instructor  knowledge  of  arbitrary  decision  rules  defined  in  these 
approaches  may  also  bias  performance  ratings. 

It  appears  that  in  actual  practice,  training  decisions  are  more  proba¬ 
bilistic  than  deterministic  judgments.  In  other  words,  instructors  and  train¬ 
ing  managers  infer  a  probability  of  a  range  of  acceptable  performance  by  the 
student  in  the  future  rather  than  making  an  absolute  prediction  of  a  specific 
level  of  performance.  The  CATES  decision  model  provides  a  method  for  assess¬ 
ing  flight  task  proficiency  based  on  the  probabilistic  nature  of  decision 
making.  Using  this  method,  an  analogy  of  the  training  program  can  be  envis¬ 
aged  as  a  biasing  process;  students  enter  the  training  program  with  a  low 
probability  of  performing  the  task  to  established  standards.  With  successive 
trials,  the  probability  of  performing  to  established  standards  Increases 
until  it  reaches  the  desired  objective  at  which  time  training  Is  terminated. 

In  summary,  the  CATES  system  promised  to  achieve  two  purposes.  First, 
it  appeared  to  offer  a  quantifiable  method  for  the  accurate  quantification 
of  student  performance  levels  needed  for  simulator  effectiveness  evaluations. 
Second,  and  perhaps  more  important,  the  CATES  system  could  provide  training 
managers  and  Instructors  with  a  valuable  tool  to  aid  the  decision-making 
process. 

MATHEMATICAL  DECISION  MODEL  USED  IN  CATES 

A  statistical  decision  model  analogous  to  determining  the  probability 
that  a  student  would  perform  a  task  to  established  standards  Is  a  sequential 
sampling  method  introduced  by  Wald  (1947)  and  described  in  Rankin  and  McDaniel 
(1980).  Appendix  A  provides  a  mathematical  discussion  of  the  Wald  Binomial 
Probability  Ratio  Test  used  as  the  statistical  decision  model.  The  sequential 
sampling  method  differs  from  conventional  sampling  methods.  Conventional 
sampling  methods  usually  require  a  fixed  number  of  Items  randomly  drawn  from 
a  larger  collection.  The  sampled  Items  are  examined  and  the  decision  is 
made  to  accept  or  reject  the  entire  collection  or  lot  based  on  this  assess¬ 
ment.  Sequential  sampling  does  not  use  fixed  sample  sizes  nor  are  the  items 
drawn  at  random  from  the  entire  lot.  Rather,  the  items  are  examined  In  the 
order  they  are  produced.  Thus,  the  sample  size  required  to  make  a  decision 
becomes  variable  and  Is  dependent  on  four  a  priori  parameters  and  the  vari¬ 
ability  of  the  ordered  sequence.  The  four  a  priori  parameters  are: 
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•  minimum  proportion  of  nondefectives  at  or  below  which  the 
collection  or  lot  is  rejected  or,  conversely,  the  proportion  of 
defectives  above  which  the  lot  is  rejected  (Pj) 

•  desirable  proportion  of  nondefectives  at  or  above  which  the 
collection  or  lot  is  accepted  (P2) 

•  risk  of  making  a  TYPE  I  decisional  error  or  declaring  the  lot 
acceptable  when  in  fact  it  Is  not  (Alpha  (a)) 

•  risk  of  making  a  TYPE  II  decisional  error  or  declaring  the  lot 
unacceptable  when  in  fact  it  is  acceptable  (Beta  (0)). 

The  variability  of  the  ordered  sequence  may  either  reduce  or  increase 
the  sample  size  required  to  make  a  decision.  For  example.  If  the  sequence 
contains  items  that  are  either  consistently  acceptable  or  consistently 
unacceptable,  a  decision  may  be  reached  with  fewer  items.  If  the  sequence 
contains  inconsistencies;  l.e.,  both  acceptable  and  unacceptable  items,  the 
sample  size  required  to  reach  the  appropriate  decision  will  increase. 

Originally,  the  sampling  procedure  was  used  to  determine  whether  a 
collection  of  a  manufactured  product  should  be  rejected  because  the  pro¬ 
portion  of  defectives  is  too  high  or  should  be  accepted  because  the  pro¬ 
portion  of  defectives  Is  below  an  acceptable  level.  In  this  industrial 
quality  control  setting,  the  Inspector  needs  a  chart  similar  to  figure  1  to 
perform  a  sequential  test  to  determine  acceptable  levels.  As  each  item  Is 
observed,  the  inspector  plots  a  point  on  the  chart  one  unit  to  the  right  If 
it  is  not  defective,  one  unit  to  the  right  and  one  unit  up  if  the  item  is 
defective.  If  the  plotted  line  crosses  the  upper  parallel  line  or  boundary, 
the  inspector  will  reject  the  production  lot.  If  the  plotted  line  crosses 
the  lower  boundary,  the  lot  will  be  accepted.  If  the  plotted  line  remains 
between  the  two  boundaries,  another  sample  item  will  be  drawn  and  observed/ 
tested.  Because  sampling  Is  expensive,  a  fixed  limit  on  the  number  of  Items 
to  be  sampled  may  be  set.  If  the  limit  Is  reached  and  the  plotted  line  has 
not  crossed  either  the  upper  or  lower  boundary,  the  inspector  must  then  make 
a  decision.  Generally,  the  decision  will  be  made  to  accept  or  reject  based 
on  the  proximity  of  the  last  plotted  point  to  the  closest  boundary  (trunca¬ 
tion).  This  decision  model  has  been  used  In  the  educational  and  training 
settings  by  Ferguson  (1970)  and  Kallsch  (1980).  Previous  use  of  the  model 
in  training  was  to  evaluate  performance  after  the  learning  period  and  to 
serve  as  an  evaluation  tool  for  computer-based  instruction  that  conserved 
testing  time  by  using  a  minimum  sample  of  Items. 

The  CATES  system  decision  model  uses  sequential  sampling  during  the 
learning  period  and  eventually  terminates  it.  Figure  2  illustrates  the  CATES 
decision  model  as  proposed  by  Rankin  and  McDaniel  (1980)  for  assessing  flight 
task  proficiency.  This  figure  shows  a  trainee  trial  sequence  of  11PPPPPP. 
Analyzing  this  sequence  using  the  decision  model  on  the  second  trial  of  the 
sequence,  the  plotted  line  would  cross  the  lower  boundary  denoting  the  student 
is  "Not  Proficient."  Thus,  the  student  Is  remediated  and  the  plot  starts 
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Figure  1.  Hypothetical  Sequential  Sampling  Chart 
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with  the  next  trial.  In  this  particular  sequence,  that  trial  Is  the  first  P 
trial  In  the  sequence.  On  the  sixth  trial  In  the  overall  sequence  (fourth 
trial  in  the  new  sequence),  the  plotted  line  crosses  the  upper  boundary  denot¬ 
ing  the  student  Is  “Proficient"  and  training  may  cease.  In  this  example, 
the  student  received  two  additional  training  trials  after  the  CATES  decision 
of  “Proficient,  Stop  Training.* 

SIMILARITY  OF  CATES  DECISION  MODEL  AND  CURRENT  OECISION  NETHOO 

The  mathematical  algorithm  used  In  the  CATES  system  closely  parallels 
the  current  decision  method  used  by  the  training  manager  or  Instructor  to 
determine  when  to  terminate  training.  Like  CATES,  the  human  judgment  method 
bases  decisions  on  varying  numbers  of  practice  trials  on  the  task  rather 
than  requiring  a  fixed  number  of  practice  trials.  Consistency  of  student 
performance  on  training  tasks  Is  also  considered  In  determining  the  appro¬ 
priate  amount  of  trials.  Students  that  perform  consistently  well  on  a  task 
are  considered  proficient  with  less  task  performance  Information  than  those 
students  that  perform  Inconsistently.  Instructors  and  training  managers 
also  appear  to  consider  the  risks  Involved  In  making  an  Inappropriate  decision. 

The  advantage  of  the  CATES  decision  model  appears  to  be  the  quantifica¬ 
tion  of  acceptable  (proficient)  performance,  unacceptable  (not  proficient) 
performance,  and  the  risks  (alpha  and  beta)  Involved  In  making  an  Inappro¬ 
priate  decision.  The  problem  then  Is  to  assess  the  advantages  offered  by 
the  mathematical  algorithm  in  Increasing  the  effectiveness  of  training  deci¬ 
sions.  The  quantifying  of  performance  and  risk  gained  through  the  use  of 
the  mathematical  algorithm  Is  an  obvious  advantage  In  training  effectiveness 
evaluations.  Other  practical  advantages  Involve  a  better  means  to  aggregate 
Inconclusive  Information  concerning  student  performance  and  a  decision  accuracy 
greater  than  the  current  method. 

ADVANTAGES  OF  MATHEMATICAL  OECISION  NDOELS 

Considerable  Investigation  has  been  conducted  on  human  decision  behavior 
and  the  cognitive  processes  humans  employ  to  make  choices  and  solve  decision- 
related  problems.  Comprehensive  reviews  of  the  experimental  literature  are 
available:  Imhoff  and  Levine  (1981),  Lee  (1971),  Nickerson  and  Feehrer  (1975), 
Rapoport  and  Wallsten  (1972),  Slovlc,  Flschhoff,  and  Lichtenstein  (1977), 
and  Slovlc  and  Lichtenstein  (1971).  Some  relevant  areas  of  study  include: 
statistical  decision  theory  (Flshbume,  1964),  game  theory  (Luce  and  Ralffa, 
1957),  and  probabilistic  Information  systems  (Edwards,  1962). 

It  Is  generally  found  that  decisions  reached  by  mathematical  models  are 
considerably  more  consistent  and  accurate  than  decisions  based  on  human  judg¬ 
ment  (Dawes  and  Corrigan,  1974;  Meehl,  1954;  Sawyer,  1966).  It  appears  that 
human  judgment  decisions  require  more  data  than  mathematical  models  as  a 
result  of  poorly  defined  parameters  and  biases  In  the  processing  of  Informa¬ 
tion  for  decisions  (Slovlc,  1976;  Tversky  and  Kahneman,  1974).  Dawes  (1979) 
proposes  that  mathematical  models  are  especially  good  at  aggregating  informa¬ 
tion  resulting  in  the  more  efficient  use  of  available  Information.  Dames 
further  suggests  that  humans  have  expertise  in  perceiving  and  sorting  Infor¬ 
mation  that  cannot  be  matched  by  a  mathematical  model. 
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Given  that  human  judgment  excels  In  perceiving  and  sorting  information 
and  that  mathematical  models  are  especially  good  at  combining  or  aggregating 
Information,  it  appears  that  a  combination  of  these  models  should  considerably 
enhance  the  decision-making  process.  It  follows  that  a  combination  of  people 
assessing  trial  performance  and  a  mathematical  model  determining  the  integra¬ 
tion  and  quantity  of  these  assessments  should  substantially  Increase  the 
validity  and  reliability  of  training  decisions.  The  potential  value  of  a 
CATES  decision  model  has  been  recognized  for  aviation  management.  Nixon 
(1981)  recommended  the  decision  model  be  used  to  assess  proficiency  of  naval 
flight  officers  undergoing  training  at  the  A-6  aircraft  Fleet  Replacement 
Squadrons. 

Although  previous  research  has  indicated  mathematical  models  may  provide 
a  potentially  valuable  decision  making  tool,  results  have  generally  been 
limited  to  laboratory  studies  and  experiments.  Evidence  is  needed  to  support 
the  practical  use  of  a  mathematical  decision  model  in  a  considerably  more 
unstructured  environment.  To  satisfy  this  need,  this  evaluation  was  conducted 
to  extend  the  knowledge  of  the  mathematical  decision  model  to  a  direct  appli¬ 
cation  in  training. 

APPLICATION  OF  THE  CATES  SYSTEM  DECISION  MODEL 

To  examine  the  practicality  of  the  CATES  system  decision  model  in  a 
realistic  training  situation,  an  evaluation  was  conducted  M1n-s1tu"  at  HS-1, 
Naval  Air  Station,  Jacksonville,  Florida.  Concurrent  with  this  study,  the 
TAEG  was  evaluating  Device  2F64C  (Browning,  McDaniel,  and  Scott,  1981; 

Browning,  McDaniel,  Scott,  and  Smode,  1982).  The  test  plan  for  this  evalua¬ 
tion  required  Instructors  to  use  the  proficiency  grading  system  to  record 
task  trial  performance  of  students  undergoing  flight  training.  As  discussed 
previously  In  this  section,  the  recording  of  task  trial  data  forms  an  Integral, 
necessary  component  of  the  CATES  system  decision  model.  In  addition  to  the 
current  method  of  making  training  decisions,  data  were  recorded  In  a  manner 
usable  by  the  CATES  system.  Although  the  proficiency  grading  system  posed 
an  additional  requirement  for  the  instructors.  It  does  not  appear  to  over¬ 
burden  them  In  accomplishing  their  duties.  Further,  most  Instructors  seem 
to  have  accepted  the  proficiency  grading  system  as  a  more  useful  method  than 
current  grading  practices. 

Rankin  and  McDaniel  (1980)  envisaged  that  full  Implementation  of  the 
CATES  system  would  require  computer  support.  Although  computer  support  Is 
available  to  HS-1  through  the  Aviation  Training  Support  System  (ATSS),  the 
TAEG  and  HS-1  agreed  that  before  using  the  ATSS  for  computer  support,  the 
efficacy  of  the  CATES  system  should  be  evaluated  to  determine  If  advantages 
could  be  realized.  If  advantages  using  CATES  were  realized,  full  Implementa¬ 
tion  could  be  Initiated. 

Full  Implementation  would  require  data  Input  to  the  ATSS.  Although 
this  may  appear  to  be  an  additional  requirement,  the  CATES  system  may  provide 
a  more  efficient  method  of  management  control  than  the  present  system  of 
maintaining  "hard  copy"  records. 

In  summary,  the  CATES  system  appears  to  place  little  additional  burden 
on  the  training  manager  than  current  methods  used  and  may  actually  relieve 
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certain  requirements.  This  is  contingent  upon  how  well  the  CATES  system 
"works"  in  the  actual  training  environment.  The  method  used  to  determine 
how  well  the  CATES  system  "works"  is  presented  in  the  next  section. 
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SECTION  III 
METHOD 


STUDENTS 

The  student  sample  consisted  of  29  newly  designated  naval  aviators  under¬ 
going  Fleet  Replacement  Pilot  Training  In  the  SH-3  aircraft  at  HS-1.  The 
students  were  resent  graduates  of  Undergraduate  Pilot  Training  at  Pensacola, 
Florida,  and  had  no  prior  flight  experience  In  the  SH-3  aircraft. 

TASKS 


The  student  was  required  to  master  approximately  190  flight  tasks  during 
Fleet  Replacement  Pilot  Training  to  become  qualified  to  fly  the  SH-3  aircraft. 
From  the  task  Inventory  of  190  tasks,  18  tasks  (appendix  B)  were  selected  to 
evaluate  the  CATES  decision  model  proposed  by  Rankin  and  McDaniel  (1980). 

These  18  tasks  were  representative  of  the  range  of  difficulty  for  tasks  In 
the  Inventory  as  well  as  tasks  Introduced  In  early  and  later  stages  of  training. 

Task  difficulty  was  determined  by  a  task  sort  Into  categories  of  "easy," 
"medium,"  and  "difficult"  and  rank  ordering  of  the  18  tasks  by  subject  matter 
experts  (HS-1  Instructor  pilots).  From  this  pool  of  18  tasks,  9  tasks  were 
selected  with  3  from  each  category  to  assess  the  efficient  use  of  Information 
needed  to  reach  a  decision.  These  nine  tasks  and  categories  are  presented 
In  appendix  C. 

INSTRUCTORS 

Flight  task  training  was  provided  by  the  28  regular  HS-1  flight  Instruc¬ 
tors.  All  Instructors  had  completed  at  least  one  tour  In  an  operational 
assignment  and  the  training  course  for  flight  Instructors  at  HS-1.  All 
Instructors  were  briefed  on  the  grading  procedures  currently  In  use  as  well 
as  the  proficiency  grading  system. 

MATERIALS  AND  EQUIPMENT 

Standard  training  materials  and  equipment  were  used  by  students  and 
Instructors  at  HS-1.  No  additional  equipment  or  materials  were  required  to 
obtain  data  and/or  Information  necessary  for  this  study.  The  primary  data  * 
collection  Instruments  were  the  standard  syllabus  grade  card  (appendix  D) 
and  the  Naval  Air  Training  and  Operating  Procedures  Standardization  Program 
(NATOPS)  Flight  Evaluation  Worksheet  (appendix  E). 

To  facilitate  retrieval  of  task  trial  Information  and  calculate  CATES 
system  decisions,  data  from  the  grade  cards  were  entered  on  a  WANG  2200  MVP 
computer  at  the  TAEG. 

PROCEDURE 

As  students  proceeded  through  the  training  syllabus,  performance  was 
graded  on  the  Syllabus  Grade  Card  using  both  current  procedures;  l.e.,  NATOPS, 
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and  the  proficiency  grading  system  procedure.  The  NATOPS  procedure  grades 
task  performance  In  three  categories  or  classifications:  "Q”  or  Qualified 
(performance  meets  or  surpasses  NATOPS  standards,  "CQ"  or  Conditionally 
Qualified  (performance  not  to  established  standards,  but  does  not  exhibit 
safety  violations),  "U"  or  Unqualified  (performance  not  to  standards  and 
safety  violations  are  exhibited).  The  NATOPS  grade  for  each  task  Is  a  sum¬ 
mary  of  all  task  trials;  l.e.,  there  Is  only  one  NATOPS  grade  for  each  task 
for  each  flight  or  session.  In  addition  to  the  NATOPS  grading  procedure, 
the  grades  for  each  practice  trial  on  each  task  were  recorded  In  the  sequence 
the  trial  was  attempted  (Proficiency  Grading  System).  Syllabus  grade  cards 
were  collected  after  each  training  flight  or  session.  Data  from  each  grade 
card  were  then  entered  Into  the  WANG  2200  MVP. 

Upon  completion  of  the  training  syllabus  and  at  the  discretion  of  the 
Instructor  pi lot/training  manager,  each  student  was  scheduled  for  a  final 
NATOPS  flight  evaluation.  The  Instructor  pi lots/training  manager  mere  not 
apprised  of  any  decisions  made  by  the  CATES  system  decision  model. 

The  NATOPS  flight  evaluation  for  each  student  was  made  by  one  of  eight 
designated  Instructor  pilots.  Flight  evaluation  grades  were  recorded  on  the 
NATOPS  Flight  Evaluation  Worksheet.  It  should  be  noted  that  the  worksheet 
does  not  specify  discrete  tasks  In  the  same  manner  as  the  syllabus  grade 
cards.  However,  If  the  student  performance  Is  below  standards  set  by  NATOPS, 
the  evaluator  Is  required  to  specify  the  task  and  explain  why  the  task  was 
not  performed  to  standards.  Thus,  performance  of  specific  tasks  on  the  flight 
evaluation  could  be  obtained.  Upon  completion,  the  NATOPS  evaluation  flight 
worksheets  were  collected.  These  worksheets  were  reviewed  and  a  determination 
was  made  concerning  the  evaluation  grade  for  each  task  and  each  student; 
l.e..  Qualified,  Conditionally  Qualified,  or  Unqualified. 

DEPENDENT  MEASURES  FOR  THE  CURRENT  DECISION  METHOD 

Two  dependent  measures  were  extracted  from  the  data  collected:  (1) 
task  performance  Information  required  to  reach  a  decision  and  (2)  the  level 
of  student  proficiency  upon  completion  of  the  training  program. 

Task  performance  Information  required  to  reach  a  training  decision  was 
determined  as  the  total  number  of  practice  trials  the  student  attempted  In 
the  flight  training  program.  Each  practice  trial  was  envisaged  as  a  "bit" 
of  Information  the  Instructors  acquired  concerning  student  performance. 

The  level  of  student  proficiency  was  determined  by  the  NATOPS  grade 
awarded  for  each  task  on  the  last  evaluation  of  training.  Grades  awarded  on 
this  basis  would  be  more  likely  to  use  the  same  standards  as  required  by  the 
NATOPS  flight  evaluation.  A  grade  of  Qualified  would  Indicate  the  Instructor 
was  confident  the  student  was  proficient  and  would  perform  the  task  to  standards 
on  the  NATOPS  evaluation.  A  grade  of  Conditionally  Qualified  would  Indicate 
the  Instructor  was  less  confident  the  student  would  perform  to  standards  and 
could  benefit  from  additional  training. 
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CATES  SYSTEM  PARAMETERS 

The  CATES  system  decision  model  requires  four  values.be  established: 

(1)  the  lowest  acceptable  proportion  of  proficient  trials  at  or  below  which 
the  student  is  considered  "Not  Proficient*  (Pi),  (2)  the  proportion  of  profi¬ 
cient  trials  at  or  above  which  represents  proficient  performance  (Pp),  (3) 
the  probability  of  a  TYPE  I  decision  error  (Alpha  or  («)),  (4)  the  probabil¬ 
ity  of  a  TYPE  II  decision  error  (Beta  or  (£)). 

The  parameter  values  were  determined  from  an  examination  of  first 
trial  performance  data  from  a  group  of  17  students  undergoing  training  at 
HS-1.  The  proportion  of  acceptable  trial  performances  to  the  total  of  first 
performances  was  used  to  set  the  Pi  value  for  each  task. 

The  P2  parameter  values  were  determined  from  the  performance  of  50  naval 
aviators  on  the  NATOPS  flight  evaluation.  The  proportion  of  Qualified  grades 
to  the  total  grades  awarded  was  calculated.  The  P9  values  were  then  estab¬ 
lished  at  one-half  standard  deviation  units  below  the  mean  proportion. 

In  the  present  study,  parameter  values  for  (a)  and  (fi)  were  arbitrarily 
selected  at  .10.  The  parameters  for  the  representative  sample  of  IS  tasks 
used  In  this  study  are  shown  In  appendix  B. 

DEPENDENT  MEASURES  FOR  THE  CATES  SYSTEM  DECISION  MODEL 

As  In  the  current  decision  method,  two  dependent  measures  were  extracted 
from  the  data  collected.  These  were:  (1)  task  performance  Information  required 
to  reach  a  decision  and  (2)  the  level  of  student  proficiency. 

Task  performance  information  required  was  determined  to  be  the  total 
number  of  practice  trials  attempted  before  a  CATES  system  decision  was  reached. 
It  Is  Important  to  note  that  because  training  and  task  practice  terminated 
at  the  discretion  of  the  Instructor  or  training  manager,  there  Is  a  possi¬ 
bility  the  CATES  system  decision  model  would  not  have  sufficient  task  trial 
Information  to  reach  a  decision.  If  the  task  protocol  had  not  resulted  In 
crossing  the  upper  boundary  of  the  decision  model  (Indicating  student  profi¬ 
ciency  was  "Undetermined"),  an  estimate  was  made  of  the  number  of  additional 
trials  required  to  make  a  decision.  This  estimate  Is  based  on  a  mathematical 
equation  developed  by  Hoel  (1971)  and  Is  shown  In  appendix  F.  Parameter 
values  used  In  this  equation  were  the  same  values  set  for  each  task.  The 
estimated  trials  to  a  decision  were  then  added  to  the  total  number  of  trials 
actually  attempted.  Using  this  procedure,  it  was  possible  for  the  CATES 
system  decision  model  to  require  either  less,  equal,  or  more  trial  Information 
to  reach  a  decision  than  the  current  decision  method  for  each  task  or  student. 

A  proficient  level  of  performance  by  the  student  was  determined  If  the 
CATES  system  decision  model  reached  a  "Proficient,  Stop  Training"  decision 
based  on  actual  trials.  Thus,  a  "Proficient,  Stop  Training"  decision  was 
considered  equivalent  to  the  current  decision  method  of  awarding  a  "Qualified" 
grade  for  the  task  on  the  last  training  flight/session. 
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CRITERION  FOR  EVALUATION  OF  OECISIONS 

The  criterion  used  to  evaluate  the  accuracy  of  the  training  decisions 
made  by  the  current  decision  method  and  the  CATES  system  decision  model  was 
the  student's  graded  task  performance  on  the  NATOPS  flight  evaluation.  If  a 
decision  concerning  proficient  level  of  performance  was  reached  and  subsequent 
task  performance  on  the  NATOPS  flight  evaluation  was  graded  as  Qualified, 
the  decisions  were  considered  correct.  If  the  grade  on  the  NATOPS  flight 
evaluation  was  either  Conditionally  Qualified  or  Unqualified,  the  decision 
was  considered  Incorrect. 

The  Information  requirements  and  accuracy  for  the  current  decision  method 
and  the  CATES  system  decision  model  were  compared.  Results  of  that  comparison 
are  described  In  the  next  section. 
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SECTION  IV 
RESULTS 

MODEL  INFORMATION  REQUIREMENTS 

The  first  analysis  dealt  with  the  amount  of  Information  required  by  the 
two  decision  methods  to  reach  a  decision  as  a  function  of  task  difficulty. 
Results  of  the  analysis  of  variance  (ANOVA)  are  shown  in  table  1. 

TABLE  1.  SOURCE  TABLE  FOR  ANOVA  OF  INFORMATION  REQUIREMENTS 

OF  TWO  DECISION  METHOOS  AND  THREE  TASK  DIFFICULTY  LEVELS 


Source 

Sum  of  Squares 

df 

MS 

F 

A  (Decision  Method) 

1250.702 

1 

1250.702 

109.81* 

Error 

318.922 

28 

11.390 

B  (Task  Difficulty) 

73.215 

1  (Adj) 

36.607 

3.22 

Error 

637.075 

43  (Adj) 

11.376 

AB  (Method  x 

Task  Difficulty) 

452.332 

2 

226.166 

43.22* 

Error 

293.058 

56 

5.233 

*P<.05 


Because  the  ANOVA  was  a  repeated  measures  design.  It  was  suspected  that 
certain  assumptions  or  requirements  of  the  ANOVA  may  have  been  violated; 
i.e.,  lack  of  homogeneity,  additivity.  A  conservative  F-test  with  reduced 
degrees  of  freedom  was  conducted  using  the  procedures  recommended  by  Ityer 
(1979).  This  conservative  F-test  still  revealed  significant  differences  for 
the  A  main  effect  (Decision  Method)  and  the  AB  Interaction  effect  (Method  x 
Task  Difficulty).  However,  for  the  B  main  effect  (Task  Difficulty),  the 
test  of  significance  failed  to  reach  the  critical  level  of  .05.  An  epsilon 
factor  (.7693)  was  determined  from  the  variance-covariance  matrix  as 
recommended  by  Greenhouse-Gel sser.  With  this  adjustment  to  the  degrees  of 
freedom,  the  B  main  effect  (Task  Difficulty)  did  not  reach  the  .05  level  of 
significance. 

To  determine  significant  differences  within  the  Interaction  effect,  the 
Tukey's  Wholly  Significant  Difference  (WSD)  was  computed.  Any  differences 
In  the  means  greater  than  2.224  may  be  considered  significant  at  the  .05 
level.  Figure  3  graphically  shows  the  relationship  between  decision  method 
and  task  difficulty  as  a  function  of  average  trials  required  to  reach  a 
"stop  training"  decision.  The  figure  shows  that  the  CATES  decision  model 
required  less  Information  to  make  a  "stop  training”  decision  across  all 
levels  of  task  difficulty.  The  Information  requirements  for  the  CATES 
decision  model  become  greater  as  task  difficulty  Increases.  Reliable 
differences  were  found  between  Information  requirements  for  easy  (x  ■  4.8 


Figure  3.  Trials  Required  to  Reach  a  "Stop  Training"  Decision  for  Two 
Decision  Methods  Across  Three  Levels  of  Task  Difficulty 
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trials)  and  difficult  tasks  (g  >8.2  trials)  assessed  by  the  CATES  model. 

For  the  human  judgment  procedure.  It  appears  the  reverse  Is  true.  More 
Information  was  collected  on  the  easy  tasks  (j  *  14.6  trials)  than  on  the 
medium  (j[  *  10.2  trials)  or  difficult  tasks  (s  *  10.5  trials).  Differences 
In  Information  requirements  for  medium  and  difficult  tasks  reached  by  human 
judgment  were  not  reliable.  The  data  Indicate  the  CATES  model  requires  less 
information  to  reach  a  decision  than  human  judgment  and  the  Information 
requirements  for  CATES  appears  to  trend  In  a  logical  manner;  i.e.,  more 
difficult  tasks  require  more  trial  Information. 

ACCURACY  OF  DECISION  METHODS 

To  determine  the  degree  to  which  the  two  decision  methods  were  able  to 
Npred1ctN  the  student's  performance  on  the  final  NATOPS  flight  evaluation 
and  compare  the  judgments  made  by  each  method  across  the  three  levels  of 
task  difficulty,  the  following  analysis  was  done.  The  judgment  made  for 
each  method  was  the  proportion  of  "Qualified"  or  "Proficient,  Stop  Training" 
decisions  to  the  overall  possible  decisions  that  could  be  made.  There  were 
87  possible  decisions  (3  tasks  X  29  students)  for  each  level  of  task  diffi¬ 
culty.  From  these  87  possible  decisions,  the  last  Instructor  grade  warded 
was  determined.  If  the  final  grade  was  a  "Q*  or  Qualified,  It  was  counted 
as  a  Qualified  judgment  made.  If  It  was  a  "CQ"  or  Conditionally  Qualified 
judgment.  It  was  not  considered  to  be  a  Qualified  judgment.  For  the  CATES 
decision  model,  only  those  student  task  protocols  that  crossed  the  upper 
boundary  resulting  in  a  "Proficient,  Stop  Training*  judgment  ware  considered 
as  a  "Qualified"  judgment.  Each  of  these  judgments  from  both  methods  were 
then  matched  against  the  task-student  evaluation  made  on  the  final  NATOPS 
flight  evaluation.  A  Qualified  judgment  made  was  considered  correct  If  a 
Qualified  grade  for  that  task  was  awarded  on  the  NATOPS  flight  evaluation. 

Table  2  shows  the  results  of  this  examination  of  the  proportion  of  Qual¬ 
ified  judgments  made  and  the  proportion  of  correct  judgments  for  the  nine 
tasks.  A  test  for  proportions  revealed  no  significant  differences  on  the 
proportion  of  qualified  judgments  made  between  decision  methods.  There  were 
no  significant  differences  found  In  the  proportions  of  correct  judgments 
made  between  methods . 


TABLE  2.  PROPORTION  OF  QUALIFIED  JUDGMENTS  AND  PROPORTION  OF 
CORRECT  DECISIONS  MADE  BY  EACH  DECISION  METHOD  ACROSS 
THREE  LEVELS  OF  TASK  DIFFICULTY 


TASK  DIFFICULTY 

_ 1 

METHOD 

Easy  (N*87) 

Medium  (N-87) 

Difficult  (N-87)  | 

Qualified 

Judgments 

Correct 

Judgments 

Qualified 

Judgments 

Correct 

Judgments 

Qualified 

Judgments 

Correct 

Judgments 

SATES 

.9885 

•  •  #OQ*f 

.5092 

.8302 

.6092 

.8112 

instructor 

.9685 

• 9001 

.7816 

.7794 

.7126 

.7097 

25 
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Although  no  significant  or  reliable  differences  were  found.  It  was  noted 
that  the  proportion  of  Qualified  judgments  made  decreased  as  task  difficulty 
Increased.  This  supports  the  Intuitive  judgment  that  the  more  difficult  or 
complex  tasks  are  somewhat  more  difficult  to  evaluate  with  confidence.  The 
CATES  decision  model  appeared  to  be  more  conservative  or  less  willing  to 
make  a  judgment  as  task  difficulty  Increased.  However,  once  a  decision  had 
been  made,  the  CATES  decision  method  tended  to  be  more  correct  than  the 
Instructor  method. 

Considering  this  trend  toward  Increased  accuracy  or  correctness  of  judg¬ 
ments  made,  the  entire  sample  of  18  tasks  was  assessed  for  Qualified  judgments 
made  and  the  accuracy  of  the  judgments.  Results  Indicated  that  for  12  of 
the  18  tasks,  CATES  was  more  correct  In  the  judgments  made.  Proportions  of 
correct  decisions  were  equal  for  the  Instructor  and  CATES  method  on  2  of  the 
18.  Instructor  judgments  appeared  to  be  more  correct  oh  4  of  the  18  tasks. 

A  sign  test  revealed  that  CATES  was  reliably  more  correct  In  judgments  than 
the  Instructors  beyond  the  .05  level  of  significance.  This  finding  would 
support  a  conclusion  that  If  CATES  decisions  were  used  to  determine  proficiency 
across  the  training  syllabus,  a  more  accurate  assessment  would  be  made  concern¬ 
ing  student  proficiency  than  the  present  method  of  Instructor  judgments. 
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SECTION  V 

DISCUSSION,  CONaUSIONS,  AND  RECOMMENDATIONS 

Results  of  this  evaluation  Indicate  the  CATES  system  decision  aodel, 
using  the  parameter  values  established  In  the  present  stu4y,  refulres  less 
Information  to  make  a  decision  than  the  current  system  of  humr  ludpnnt. 
Decisions  reached  by  the  CATES  system  reflected  a  higher  prop  Jn  of  correct 
decisions  in  reference  to  the  NATOPS  flight  evaluation.  Acros*  representa¬ 
tive  sample  of  18  tasks,  the  CATES  system  was  reliably  more  accwrate  than 
the  current  method  of  human  judgment.  The  finding  that  the  CATES  system 
requires  less  trial  information  to  reach  a  decision  of  greater  precision 
strongly  supports  the  CATES  system  decision  model's  superior  efficiency  when 
compared  to  the  current  method  of  making  training  judgments.  Results  of 
this  study  extend  previous  research  results  suggesting  greater  consistency 
and  accuracy  of  mathematical  models  to  an  actual  training  situation  In  a 
considerably  more  unstructured  environment. 

The  proportion  of  judgments  concerning  student  proficiency  for  easy 
tasks  was  high  and  equal  for  both  methods.  As  task  difficulty  Increased, 
however,  the  CATES  system  model  made  a  lower  proportion  of  decisions  than 
the  current  method  of  Instructor  judgments.  The  conservatism  or  riskiness 
of  the  CATES  system  model  Is  established  through  parameter  values,  specific¬ 
ally  alpha  (a)  and  beta  (/3).  Since  these  parameter  values  were  held  constant 
across  all  tasks  and  levels  of  task  difficulty.  It  Is  reasonable  to  conclude 
the  Instructors  were  willing  to  take  more  risks  In  decisions  made  on  medium 
and  difficult  tasks.  This  willingness  to  take  greater  risks  may  result  In 
the  lowered  proportions  of  correct  decisions  made  by  the  Instructors.  Results 
of  this  study  are  similar  to  a  study  of  human  decision  making  behavior  In  a 
sequential  testing  situation  reported  by  Becker  (1958).  According  to  Becker, 
subjects  appeared  to  operate  more  like  Wald's  sequential  sampling  model  when 
the  problem  was  difficult  than  when  the  problem  was  easy.  Typically,  subjects 
required  relatively  more  samples  or  Information  on  easy  problems  and  relatively 
less  information  on  the  difficult,  as  If  they  set  alpha  (a)  and  beta  (0) 
lower  for  the  easy  problems. 

The  reasons  why  Instructors  obtained  considerably  more  Information  than 
the  CATES  system  required  for  easy  tasks  remains  unclear.  This  may  have 
resulted  from: 

•  easy  tasks  being  Introduced  earlier  In  the  training  program  allowing 
more  time  for  practice 

•  easy  tasks  being  prerequisite  to  the  performance  of  the  more  diffi¬ 
cult  tasks;  e.g.,  normal  starting  of  the  engines  were  required  to 
accomplish  more  difficult  flight  tasks 

•  Instructors  allowing  students  to  perform  easy  tasks  so  that  success¬ 
ful  performance  would  motivate  the  student  to  perform  better  on 

the  more  difficult  tasks 

•  Instructors  being  reinforced  by  the  student's  demonstrated  high 
levels  of  performance  on  the  easy  tasks  thus  Increasing  the 
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probability  the  Instructor  will  request  the  student  to  perform  the 
task  again 

•  Instructors  obtaining  Information  about  student  performance  of 
easy  tasks  Is  done  at  a  lower  "cost."  Easier  tasks  are  probably 
less  complex  to  evaluate  and  do  not  require  as  high  a  degree  of 
actual  physical  risk  to  the  student  and  Instructor  than  more  diffi¬ 
cult  complex  tasks  present. 

Whether  there  Is  single  or  multiple  causes.  It  would  appear  that  easy 
tasks  are  "overl earned"  as  a  result  of  significantly  more  practice  allowed. 
This  overlearning  probably  results  In  considerable  performance  consistency 
by  the  student  resulting  in  high  agreement  between  the  current  decision 
methods,  the  CATES  system  decision  model,  and  the  NATOPS  flight  evaluation. 
Considering  the  greater  consistency  of  performance  and  the  high  agreement 
between  decision  and  evaluation  method.  It  appears  that  overlearning  Is 
highly  desirable. 

The  more  salient  Issue  from  the  training  managers  point  of  view  concerns 
the  cost  of  "overtraining."  The  results  Indicated  the  amount  of  training 
provided  for  easy  tasks  in  excess  of  that  required  to  make  a  CATES  system 
decision;  however.  It  was  not  within  the  scope  of  this  study  to  determine 
the  economic  or  training  costs  Incurred  by  training  beyond  acceptable  profi¬ 
ciency  levels.  If  such  an  evaluation  were  conducted  In  the  future.  It  would 
be  necessary  to  consider  several  possible  causes  of  "overtraining"  rather 
than  simply  the  amount  and  cost  of  providing  training  beyond  required  Vevels. 

Of  considerable  Interest  to  the  training  manager  Is  the  Issue  "under- 
training"  the  medium  and  difficult  tasks.  Neither  the  CATES  syst&a  nor  the 
current  human  judgment  method  were  able  to  render  qualified  or  proficient 
judgments  In  20  to  40  percent  of  the  proficiency  decisions.  A  paradox  seems 
to  exist  In  the  data.  While  the  CATES  system  decision  mndel  appeared  to  be 
more  conservative  In  making  a  judgment  than  the  current  decision  method,  the 
amount  of  trial  Information  needed  to  reach  a  decision  was  reliably  less  for 
the  CATES  decision  model.  It  would  appear  logical  that  a  relatively  conser¬ 
vative  method  would  require  more  data  or  task  performance  Information.  Train 
Ing  trial  sequences  were  Individually  examined  to  determine  reasons  for  this 
apparent  paradox.  The  observation  was  made  that  students  demonstrating  con¬ 
sistent  proficient  performance  continued  to  perform  training  trials  well 
after  the  CATES  decision  (overlearning).  Conversely,  students  with  more 
variable  task  protocols  were  not  afforded  the  opportunity  to  practice  the 
task  with  a  sufficient  number  of  trials  needed  to  reach  a  CATES  decision. 

It  would  appear  that  the  paradox  of  the  more  conservative  model  requiring 
less  Information  to  make  a  decision  could  be  attributed  to  under  and  over¬ 
training  In  the  medium  and  difficult  tasks. 

An  Important  methodological  restriction  was  placed  on  this  evaluation. 
Students  proceeded  through  the  training  program  at  the  discretion  of  the 
Instructor /training  manager  under  the  current  decisional  method.  In  the 
event  the  CATES  system  reached  a  decision,  training  may  have  continued. 
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Although  in  a  strict  sense,  the  CATES  system  would  consider  the  additional 
task  training  and  trial  information  unnecessary  to  reach  a  decision,  in  a 
practical  sense  this  additional  training  may  have  been  an  important  factor 
in  the  final  NATOPS  flight  evaluation.  Certainly  no  Implication  should  be 
made  that  training  trials  beyond  a  CATES  system  decision  of  proficiency  is 
unnecessary  overtraining.  The  next  logical  step  in  eliminating  this  method¬ 
ological  flaw  would  be  to  further  evaluate  the  CATES  system  with  methodology 
similar  to  that  used  in  the  present  study,  with  the  exception  that  the  pro¬ 
cedure  should  provide  for  additional  training  beyond  the  current  decision 
method  if  additional  information  is  required  to  reach  a  CATES  system  decision. 
Thus,  the  proportion  of  proficient  judgments  made  by  the  CATES  system  would 
increase.  If  results  were  found  similar  to  this  study,  strong  evidence  would 
be  available  for  employing  the  CATES  system  in  an  important  role  for  training 
decisions. 

It  should  be  noted  that  the  criterion  measure  for  both  the  current  deci¬ 
sion  method  and  the  CATES  system  decision  model  was  performance  on  the  NATOPS 
flight  evaluation.  Although  the  NATOPS  flight  evaluations  are  conducted 
using  specially  selected,  experienced  naval  aviators,  no  measures  of  validity 
or  reliability  have  been  determined  for  that  procedure.  Essentially,  perfor¬ 
mance  on  the  NATOPS  flight  evaluation  is  determined  in  the  same  manner  as 
used  by  the  current  decision  method  on  training  flights.  The  fact  that  NATOPS 
evaluators  are  specially  selected,  experienced,  and  trained  may  very  well 
result  in  a  greater  reliability  for  the  NATOPS  evaluation  of  flight  perfor¬ 
mance.  Nevertheless,  It  Is  still  subject  to  the  problems  of  human  variabil¬ 
ity;  i.e.,  biases,  varying  standards,  personal  Interaction.  Determination 
of  the  validity  and  reliability  of  criterion  measures  Is  a  difficult  and 
elusive  task.  However,  If  naval  aviation  continues  to  use  the  NATOPS  flight 
evaluation  as  a  yardstick  to  measure  flight  performance,  it  Is  desirable 
that  this  task  be  undertaken. 
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CONCLUSIONS  AND  RECOMMENDATIONS 

Based  on  the  results  of  this  study, 
recommendations  are  made. 

CONCLUSION 

The  CATES  system  Is  particularly 
useful  to  manage  the  training 
syllabus  at  the  lowest  element  of 
the  syllabus  (flight  task). 

Changes  to  the  syllabus  resulting 
from  addltlon/deletlon/modlflca- 
tion  of  the  flight  tasks  can  be 
made  quickly  and  efficiently. 

The  determination  of  student  per¬ 
formance  at  the  task  level  rather 
than  event /sesslon/f light  level 
provides  a  more  well  defined  pic¬ 
ture  of  student  performance.  It 
allows  the  Instructor /training 
manager  to  determine  student 
strengths  and  weaknesses  In  a 
more  timely  manner. 

The  Proficiency  Grading  System 
provides  student  task  performance 
Information  with  better  defini¬ 
tion  and  specificity  than  the 
currently  used  NATOPS  grading 
procedures. 


The  CATES  system  decision  model 
appears  to  be  more  efficient  and 
accurate  than  the  current  method 
of  determining  student  task  pro¬ 
ficiency  and  making  training 
judgments. 

Method  used  for  establishing 
CATES  system  model  parameters; 
i.e..  Pi,  P2,  (a),  and  (/8), 
appears  to  be  reasonable  and  In 
general  agreement  with  the 
present  system  of  making  training 
decisions. 


Data  from  this  study  Indicate 
considerable  variability  In 
Instructor  judgments.  Levels  of 


the  following  conclusions  and 


REC0MCNDATI0N 

HS-1  should  consider  extending 
the  current  ATSS  of  managing  the 
syllabus  at  the  event  level  to 
tasks  trained  within  each  event. 


HS-1  should  focus  on  student  per¬ 
formance  of  Individual  tasks  In 
the  training  syllabus.  Capabili¬ 
ties  of  the  ATSS  to  record  student 
performance  on  events/sessions/ 
flights  should  be  extended  to 
record  student  task  performance 
within  an  event/sesslon/fllght. 


HS-1  should  continue  to  use  the 
proficiency  grading  procedures 
for  Category  I  replacement  pilots. 
The  proficiency  grading  procedure 
should  be  extended  to  Include  all 
categories  of  replacement  pilots. 


Based  on  positive  results  of  future 
evaluations,  HS-1  should  consider 
Incorporating  the  CATES  system 
decision  model  to  augment  the 
current  method  for  owing  training 
decisions. 

Continue  using  this  method  to 
establish  parameters  for  all  tasks 
to  be  trained  In  the  replacement 
pilot  training  syllabus. 


HS-1  should  continually  train  and 
standardize  Instructors  to  reduce 
variability  In  grading  student 
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risk  appear  to  vary  with  task 
difficulty  and  instructors  when 
using  the  NATOPS  grading  procedures. 
This  variability  and  instructor 
bias  may  affect  the  reliability 
of  the  NATOPS  grading  procedure 
to  a  considerable  degree. 

The  CATES  system  decision  model 
is  useful  to  preclude  the  “under- 
training"  of  tasks. 


The  CATES  system  decision  model 
may  be  useful  to  determine  exces¬ 
sive  task  training  (overtraining). 


The  CATES  system  could  be  adapted 
to  other  FRS  flight  training 
programs. 


The  CATES  system  may  provide  a 
more  efficient  and  accurate  method 
of  determining  student  performance 
in  Undergraduate  Pilot  Training. 


The  validity  or  reliability  of 
the  NATOPS  flight  evaluation  has 
not  been  determined.  Hie  evalua¬ 
tion  is  subject  to  the  same 
vagaries  and  variability  noted  In 
evaluating  student  performance  In 
the  training  program. 


performance,  thus  increasing  the 
reliability  of  the  grading. 


Student  task  performance  should 
be  evaluated  using  the  CATES  deci¬ 
sion  model.  If  proficiency  level 
cannot  be  determined  by  the  CATES 
system  within  parameters  used  by 
the  system,  training  should  be 
continued  until  a  decision  is 
reached. 

Tasks  that  are  trained  beyond 
levels  required  by  the  CATES 
system  decision  model  should  be 
carefully  monitored  to  ensure  the 
additional  training  Is  desirable 
for  improving  student  performance 
across  the  overall  flight  syllabus. 

If  subsequent  evaluations  reveal 
the  CATES  system  continues  to 
result  In  greater  efficiency  and 
higher  accuracy  In  reaching  train¬ 
ing  decisions,  other  FRSs  may 
consider  incorporating  the 
CATES  system  Into  their  training 
programs. 

If  subsequent  evaluations  reveal 
the  CATES  system  continues  to 
result  In  greater  efficiency  and 
higher  accuracy  in  reaching  train¬ 
ing  decisions,  the  Chief  of 
Naval  Air  Training  should  con¬ 
sider  evaluating  the  CATES  system 
for  possible  Inclusion  In  Under¬ 
graduate  Pilot  Training. 

Naval  Air  Systems  Command  should 
consider  Initiating  a  program  to 
determine  the  validity  and  reli¬ 
ability  of  the  NATOPS  flight 
evaluation  program. 
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POST  NOTE 

This  study  provides  evidence  that  a  mathematical  decision  model*  specif¬ 
ically  the  CATES  system  decision  model*  can  powerfully  augment  present  train¬ 
ing  decision  methods  for  replacement  pilots  undergoing  training  at  the  FRS. 

It  1$  worthy  to  note  that  In  addition  to  achieving  more  accurate  and  precise 
training  decisions,  the  CATES  system  also  provides  a  useful  tool  for  the 
management  of  a  curriculum.  The  CATES  system  provides  documentation  as  well 
as  student  performance  measures  at  the  lowest  level  or  element  in  the  curric¬ 
ulum;  l.e.*  the  flight  task.  This  documentation  and  recordkeeping,  combined 
with  the  apparent  effective  tool  for  making  training  decisions,  makes  the 
CATES  system  especially  amenable  as  a  computer-based  or  computer-managed 
Instructional  system. 

As  a  result  of  this  conceptual  logic  and  the  findings  In  this  study* 

HS-1  Is  aggressively  pursuing  the  Incorporation  of  the  CATES  system  Into  the 
ATSS  to  aid  In  Increasing  the  efficiency  of  training  management.  Upon  com¬ 
pletion  of  this  effort.  It  Is  envisaged  that  the  procedures  used  In  incorpo¬ 
rating  the  CATES  system  Into  the  ATSS  accompanied  by  a  user's  manual  will  be 
published  In  a  future  TAEG  report. 

In  addition*  further  evaluation  of  the  CATES  system  decision  model  Is 
being  planned  at  HS-1  to  provide  additional  training  required  to  reach  a 
CATES  system  decision.  Such  an  evaluation  will  extend  the  findings  In  this 
study  by  providing  actual  rather  than  estimated  Information  required  to  reach 
a  decision.  This  planned  evaluation  will  also  determine  If  the  additional 
training  will  Impact  on  the  NATOPS  flight  evaluation  In  terms  of  accuracy 
and  precision  of  decisions  similar  to  the  findings  In  this  study.  Results 
of  this  study  will  also  be  published  as  a  TAEG  report. 
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APPENDIX  A 

WALD  BINOMIAL  PROBABILITY  RATIO 
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WALD  BINOMIAL  PROBABILITY  RATIO  TEST 

The  Wald  binomial  probability  ratio  test  was  developed  by  Wald  (1947)  as 
a  means  of  making  statistical  decisions  using  as  limited  a  sample  as  possible 
The  procedure  Involves  the  consideration  of  two  hypotheses: 

and  H^:  P  2  Pg  where 

P  Is  the  proportion  of  nondefectives  In  the  collection  under  consideration, 

P]  Is  the  minimum  proportion  of  nondefectives  at  or  below  which  the  collec¬ 
tion  Is  rejected,  and  P2  Is  the  desired  proportion  of  nondefectives,  at  or 
above  which  the  collection  Is  accepted.  Since  a  simple  hypothesis  Is  being 
tested  against  a  simple  alternative,  the  basis  for  deciding  between  H  and 
H.j  may  be  tested  using  the  likelihood  ratio: 

(P;)d"  (1  -  P2)n~dl1 

P,n  (P,)""  0-  P,)  "'dn 

Where:  P.  -  Minimum  proportion  of  nondefectives  at  or  below  which  the 

collection  Is  rejected. 

P2  -  Desirable  proportion  of  nondefectives  at  or  above  which  the 
collection  Is  accepted. 

n  «  Total  Items  In  collection. 

dn  *  Total  nondefectives  In  collection. 

The  sequential  testing  procedure  provides  for  a  postponement  region 
based  on  prescribed  values  of  alpha  (a)  and  beta  (3)  that  approximate  the 
two  types  of  errors  found  In  the  statistical  decision  process.  To  test  the 
hypothesis  HQ:  P  ■  Pj,  calculate  the  likelihood  ratio  and  proceed  as  follows: 

1.  If  **2n  <  3 ,  accept  H. 

pln 

p  . 

2.  2n  ^  1- g .  accept  H-j 

*ln  a 

p 

3.  If  3  K  2n  <  l-£ .  take  an  additional  observation. 

T5T 

These  three  decisions  relate  well  to  the  task  proficiency  problem.  We 
may  use  the  following  rules: 

1.  Accept  the  hypothesis  that  the  grade  of  P  Is  accumulated  In  lower 
proportions  than  acceptable  performance  would  Indicate. 
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2.  Reject  the  hypothesis  that  the  grade  of  P  Is  accumulated  In  lower 
proportions  than  acceptable  performance  would  Indicate.  By  rejecting  this 
hypothesis,  an  alternative  hypothesis  Is  accepted  that  the  grade  of  P  is 
accumulated  in  proportions  equal  to  or  greater  than  desired  performance. 

3.  Continue  training  by  taking  an  additional  trial(s);  a  decision 
cannot  be  made  with  specified  confidence. 

The  following  equations  are  used  to  calculate  the  decision  regions  of 
the  sequential  sampling  decision  model. 


dn  £ 


log  p2  +  log  ^1 


log  ^l 


log  P2  +  log  1-pl 


dn  ^ 


log  1-fl 

a 

log  p2  +  log  ll^l 

-PT  1^2 


log  !-pl 


log  P2  +  log  1’pl 


Where:  dn  *  Accumulation  of  trials  graded  as  "P"  In  the  sequence 

n  »  Total  trials  presented  In  the  sequence 

. 

P1  ■  Lowest  acceptable  proportion  of  proficient  trials  (P)  required 
to  pass  the  NATOPS  flight  evaluation  with  a  grade  of  "Qualified.1 

P2  ■  Proportion  of  proficient  trials  (P)  that  represent  desirable 
performance  on  the  NATOPS  flight  evaluation. 


Alpha( «)  -  The  probability  of  making  a  type  I  error  (deciding  a  student  Is 
proficient  when  In  fact  he  Is  not  proficient). 

Beta(£)  =  The  probability  of  making  a  type  II  error  (deciding  a  student 
Is  not  proficient  when  In  fact  he  Is  proficient). 

Hie  first  term  of  the  two  equations  will  determine  the  Intercepts  of  the 
two  linear  equations.  The  width  between  these  Intercepts  Is  determined 
largely  by  values  selected  for  alpha  (a)  and  beta  08).  The  width  between  the 
Intercepts  translates  Into  a  region  of  uncertainty;  thus  as  lower  values  of 
alpha  (a)  and  beta  (0)  are  selected  this  region  of  uncertainty  Increases. 
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The  second  term  of  the  equations  determines  the  slopes  of  the  linear 
equation.  Since  the  second  term  Is  the  same  for  both  equations,  the  result 
will  be  slopes  with  parallel  lines.  Values  of  Pi  and  ?2  as  well  as  differences 
between  P,  and  P2  affect  the  slope  of  the  lines.  This  Is  easily  translated 
Into  task  difficulty.  As  Pg  values  Increase,  Indicating  easier  tasks,  the 
slope  becomes  more  steep.  This  In  turn  results  in  fewer  trials  required  In 
the  sample  to  reach  a  decision. 

As  differences  In  Pi  and  ?2  Increase,  the  slope  also  becomes  steeper  and 
the  uncertainty  region  decreases.  This  Is  consonant  with  rational  decision 
making.  When  the  difference  between  the  lower  level  of  proficiency  and  upper 
level  of  proficiency  Is  great.  It  Is  easier  to  determine  at  which  proficiency 
level  the  pilot  trainee  Is  performing.  The  concept  of  differences  In  Pi 
and  ?2  Is  analogous  to  the  concept  of  effect  size  In  statistically  testing 
the  difference  between  the  means  of  two  groups.  In  such  statistical  testing, 
when  alpha  (a)  and  beta  (jB)  remain  constant,  the  number  of  observations 
required  to  detect  a  significant  difference  may  be  reduced  as  the  anticipated 
effect  size  Increases  (Kallsch,  1980). 
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APPENDIX  B 

TASKS  AND  PARAMETER  VALUES  USED  IN  EVALUATION 
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TASKS  AND  TASK  PARAMETERS  USED  IN  EVALUATION 


Parameters 

Task  Description 

Alpha(a) 

Beta(0) 

Pi 

P2 

1.  Normal  Landing 

.10 

.10 

.18 

.78 

2.  Normal  Approach 

.10 

.10 

.18 

.78 

3.  Free  Stream  Recovery 

.10 

.10 

.12 

.55 

4.  Single  Engine  Approach 

.10 

.10 

.18 

.75 

5.  Single  Engine  Landing 

.10 

.10 

.53 

.75 

6.  Single  Engine  Malfunction 
Analysis 

.10 

.10 

.06 

.51 

7.  ASE  Off  Landing 

.10 

.10 

.30 

.86 

8.  Alternate  Approach  Pilot 
Procedures 

.10 

.10 

.18 

.69 

9.  Windline  SAR  Pilot  Procedures 

.10 

.10 

.38 

.80 

10.  Normal  Start 

.10 

.10 

.12 

.65 

11.  Rotor  Engagement 

.10 

.10 

.47 

.75 

12.  Single  Engine  Malfunction 
Takeoff  Abort 

.10 

.10 

.41 

.75 

13.  Automatic  Approach  Pilot 
Procedures 

.10 

.10 

.24 

.90 

14.  Servo  Malfunction 

.10 

.10 

.25 

.62 

15.  Manual  Throttle 

.10 

.10 

.35 

.51 

16.  ASE  Malfunction 

.10 

.10 

.35 

.62 

17.  SAR  Manual  Approach 

.10 

.10 

.25 

.80 

18.  Shutdown  Checklist 

.10 

.10 

.29 

.91 
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TASKS  AND  LEVEL  OF  DIFFICULTY  USED  TO  EVALUATE  EFFICIENCY 


Level  of  Difficulty 

Tasks 

Easy 

Normal  Start 

Shutdown  Checklist 

Normal  Landing 

Medium 

SAR  Manual  Approach 

Alternate  Approach  Pilot  Procedures 

Single  Engine  Malfunction  Takeoff  Abort 

Difficult 

Wlndllne  Search  and  Rescue  Pilot  Procedure 
ASE  Off  Landing 

Freestream  Recovery 
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H-3  PILOT  NATOPS  EVALUATION  WORKSHEET 
(Rtv.  9-79) 


PILOT 

PLIGHT  OATS  _ GRADE 
FLIGHT  DURATION  _____________________  BONO 

SIDS  HOMER  _ 

OPEN  BOOH  EXAM  OATS  _  GRADE 

CLOSED  BOOK  EXAM  OATS  ' _  GRADE 

ORAL  EXAM  OATS  _  GRADE 


OVERALL  PINAL  GRADS 


EVALUATOR 


1.  A  grade  of  unqualified  in  any  critical  area/sub  araa 
will  result  in  an  overall  grade  of  unqualified  for  the 
flight. 

2.  A  grede  of  conditionally  qualified  in  a  critical 
will  result  in  an  overall  grade  of  conditionally 
qualified  for  the  flight. 

3.  Only  the  musters  0,  2.  or  4  will  be  assigned  to  a 
areas.  No  interpolation  is  allowed. 


Unqualified . . .  0.0 

Conditionally  Qualified  .  2.0 

Qualified  . 4.0 
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1.  Electrical  Malfunctions. 

a.  Generators 

b.  Electrical  fire 

2.  ASE  Malfunctions 

a.  Pitch 

b.  Roll 

c.  Collective 

d.  Yaw 

3.  Transmission  Malfunctions 
•a.  Chip  detected 

*b.  Pressure  loss 

c.  Tail  takeoff 

d.  Torque  system 

4.  Engine  Malfunctions 
*a.  Engine  fire 

*b.  Flex  shaft 

c.  Oil  pressure 

d.  Oil  temperature 

e.  Hot  start 

f.  Post  shutdown  fire 

g.  PMS 

5.  Rotary  Rudder  Malfunctions 
•a.  Tail  Rotor  control/drive  loss 
•b.  TGB/IGB  chip  light 

6.  Fuel  System  Malfunctions 

a.  Fuel  filter  bypass 

b.  Fuel  boost  pump 

7.  Hydraulic  Malfunctions 

a.  Primary 

b.  Auxiliary 

c.  Utility 

d.  Sensing  unit 

8.  Water  Operations 
*a.  Water  landing 

b.  Water  takeoff 

c.  Fuel  dumping 

9.  Rotor  Brake  Malfunctions 

a.  Inflight 

b.  Shutdown 
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PILOTS  ORAL  EMERGENCY  WORKSHEET 


GRADE 


10.  Discussion  Items 
*a.  Power  settling 
*b.  Blade  stall 
*c.  Dynamic  rollover 

d.  Sonar  hoist 

e.  MAD  reeling  machine 

f.  AKT  22  antenna 


GENERAL  COMMENTS 


OVERALL GRADE. 
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PILOT  EVALUATIONS  WORKSHEET 


Area  I. 

Ground  Operations 

— 

a. 

Brief/debrief/flight  gear 

b. 

Records  check 

c. 

Preflight/postflight 

• 

d. 

Checklist  procedures/systems  check 

e. 

Start /engagement 

__ 

f. 

Taxi/lookout 

* 

g- 

Disengagement/shutdown 

_ m 

h. 

General 

Areal.  Ground  Operations 

CONDITIONALLY  QUALIFIED.  Did  not  fully  instruct  or  debrief  the  crew. 
Flight  equipment  improperly  worn  or  in  marginal  condition.  Did  not  fully 
examine  flight  records.  Minor  omissions  or  errors  on  preflight  or  postflight. 
Improper  or  incomplete  use  of  checklists.  Non  standard  procedures. 
Inattention  or  misinterpretation  of  visual  signal.  Rough  or  erratic  start, 
engagement,  disengagement  or  shutdown. 

UNQUALIFIED.  Did  not  conduct  brief  or  debrief.  Flight  equipment  miss¬ 
ing,  not  worn  or  in  an  unsafe  condition.  Failed  to  sign  for  aircraft  or 
accepted  aircraft  with  grounding  discrepancy.  Failed  to  note  or  record 
downing  discrepancy  after  flight.  Any  omission  or  error  on  preflight  or 
postflight  which  would  affect  safety  of  flight.  Exceeded  published  limita¬ 
tions  during  start,  engagement,  disengagement  or  shutdown.  Did  not 
utilize  checklist  or  perform  required  systems  checks.  Marginal  control  of 
.helicopter  while  taxiing.  Ignored  visual  signal.  Did  not  use  pre-takeoff 
checklist. 
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Ana  II.  Normal  Flight  Operation* 

a.  Checklist  procedures 

b.  Transit! on /climb 

c.  Cruise  flight 

d.  Systems  knowlcdge/usage 
t:  Normal  landings/takeoffs 

f.  Hover  Aow  work 

g.  General 


Area  11.  Normal  Flitfit  Operations 

CONDITIONALLY  QUALIFIED.  Incomplete  use  of  takeoff,  post-takeoff, 
or  landing  checkliat.  Application  of  power  erratic  but  did  not  exceed 
limitations.  Unable  to  maintain  altitude  within  tSO  feet  of  assigned 
altitude.  Maintained  airspeed  within  ±10  knots.  Heading  control  varied  ±5 
degrees  between  final  approach  and  landing.  Hover  altitude  1 S  feet  ±S  feet. 

Unable  to  fully  explain  aircraft  systems  or  limitations. 

UNQUALIFIED.  Did  not  use  checklist.  Did  not  check  instruments  prior  to 
leaving  hover.  Failed  to  use  sufficient  power  or  exceeded  aircraft  or  engine 
limitations.  Safety  precautions  not  observed.  Leveled  off  in  excess  of  SO 
feet  from  assigned  altitude.  Airspeed  tolerance  ±10  knots  excceded»Hoverin  excess 
I S  ±S  feet,  excessive  nose  attitude  or  lateral  drift  on  touchdown.  Running 
(andings/takeoffs  in  excess  of  40  knots,  yaw  in  excess  of  10  degrees  or 
lateral  drift  on  touchdown 'takeoff.  Unsatisfactory  knowledge  of  aircraft 
systems  or  limitations. 
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Area  111.  Emergency  Operations 

a.  Autorotations. 

b.  Single  engine  landings/waveoff 

c.  AUX  off  landings 

d.  ASE  off  landings/takeoffs 

e.  Emergency  procedures 

f.  General 


Area  III.  Emergency  Operations 

CONDITIONALLY  QUALIFIED.  Did  not  pre-brief  co-pilot  on  autorota¬ 
tions.  Airspeed,  Nr  and  heading  control  erratic.  Groundspecd  exceeded  15 
knots  or  slight  drift  at  recovery.  Did  not  establish  and  maintain  minimum 
safe  single  engine  speed  on  landings  or  waveoffs.  Minor  difficulty  in 
controlling  Nr  during  single  engine.  Power,  heading  and  altitude  control 
erratic  during  AUX  or  ASE  off  flight.  Did  not  fully  comply  with  emergency 
procedures  bud  did  not  jeopardize  aircraft  or  crew. 

UNQUALIFIED.  During  autorotation  did  not  call  for  full  power.  Airspeed, 
Nr  and  heading  control  beyond  safe  limits.  Implemented  techniques  that 
would  have  jeopardized  the  successful  completion  and  recovery  of  the  auto¬ 
rotation.  Failed  to  call  for  full  power/PMS  off  during  single  engines.  Failed 
to  note  or  correct  low/unsafe  Nr  conditions  during  single  engine.  Exceeded 
rate  of  descent  limits  during  single  engine  approach  or  engine  limits.  ASE 
off  and  AUX  off  flight  unsafe  or  excessive  lateral  drift/rate  of  descent  on 
touchdown.  Failed  to  comply  with  established  emergency  procedures  which 
resulted  in  jeopardizing  aircraft/crew  or  exceeded  cnginc/airframc  limitations. 
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Area  IV.  Coupier/Sonar  Operations  (Hooded) 

a.  Checklist/voice  procedures 

b.  Automatic  approach 

c.  Alternate  approach 

d.  Climb  out 

e.  Coupler/sonar  emergencies 

f.  Systems  knowledge, usage 

g.  General 


Area  IV.  Coupler/Sonar  Operations  (Hooded) 

CONDITIONALLY  QUALIFIED.  Minor  deviations  from  established  check 
list  and  voice  procedures.  Erratic  control  of  aircraft  during  automatic, 
alternate  approach  and  climb  out.  Erratic  altitude  control  of  ISO  feet  ±20 
feet.  Reacted  slowly  to  emergencies.  Unable  to  fully  explain  systems  or 
limitations. 

UNQUALIFIED.  Checklist  not  used  or  unsafelimproper  procedures 
utilized.  Allowed  aircraft  to  descend  through  30  feet  in  hover  without 
attempting  to  correct.  Made  omissions  or  errors  in  emergency  procedures 
that  could  jeopardize  aircraft  or  crew.  Attempted  to  hover  downwind 
without  correcting.  Unsatisfactory  knowledge  of  systems  or  procedures. 
Unable  to  consistently  maintain  )  SO  ±30  feet  while  hooded. 
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Area  V.  Search  and  Rescue  Operations 

a.  Navigation 

b.  IFR  procedures  (Hooded) 

c.  VFR  procedures 

d.  Crew/cockpit  coordination 

e.  General 


\rca  V.  Search  and  Rescue  Operations  (Hooded) 

CONDITIONALLY  QUALIFIED.  No  coordination  of  visual  lookout 
doctrine.  Used  nonstandard  voice,  approach,  pattern  or  hoist  proce¬ 
dures  but  none  which  would  seriously  affect  the  mission.  Did  not  fully 
properly  utilize  copilot/crew  and  systems  in  accomplishing  rescue. 

UNQUALIFIED.  Could  not  follow  wind  line  rescue  pattern.  Hovered 
downwind  without  correcting.  Unable  to  consistently  maintain  1 10  ±30 
feet  hooded.  Allowed  aircraft  to  descend  below  30  feet  during  approach; 
hover  without  correcting.  Exceeded  aircraft  limitations  or  procedures  that 
would  have  jeopardized  aircraft  or  crew. 
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MATHEMATICAL  EQUATION  USED  TO  ESTIMATE  TRIALS  TO  A 
"PROFICIENT,  STOP  TRAINING"  DECISION 


j8  log  — +  1-/3  log  JbjB. 


Additional  Estimated  Trials  to  ?z 


i-p2  p2 

1  -  P2  log  +  ?9  log - 

1-P1  P! 


Overall  Estimated  Trials  *  Trials  Performed  by  Student  +  Additional 
Estimated  Trials  to  ?z  (estimated  trials  required  to  cross 
the  upper  boundary) 
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